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ABSTRACT 

The Illinois Three Phase System should be temporarily 
retained as an administrative tool to be used in relationships 
between the Division of Vocational and Technical Education and local 
education agencies. But it is clear that such a system, with its 
scope directed toward local school districts, with its lack of 
outcome assessment, and with irs lack of state priorities should not 
be judged a fully satisf ? -^tory vehicle for statewide evaluation. The 
greatest need for improveuient and modification of the System is for 
improved, fully comprehensive follow-up procedures. Similarly^ there 
is a great need for progress .la the area of performance assessment 
for students who complete programs. A supplemental developmental 
program is needed, which should lead to progressive modification of 
the present System with the goal of ultimately replacing it. 
(Author/CK) 
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OVERVIEW OF STUDY OUTCOMES 



The primary concern of this study was to determine the efficiency 
and efficacy of evaluation practices of the Illinois Divisiion of Vocational 
and Technical Education (DVTE).* Its priniary focus has been on the perform- 
ance of the Three Phase System for Statewide Evaluation Occupational Pro- 
grams administered by the Program Approval and Evaluaton Unit of the Division. 

Study of the System was conducted by use of several methodologies: (1) 
study of the many documents prepared by thr DVTE, (2) interviewi3 of Division 
personnel, (3) sampling of agencies for study, (4) collection of data from re- 
cord systems of the Division, (5) submission of questionnaires to planners and 
team members, and (6) independent study of trends in statewide ev«^luation. 

The Three Phase System has been operated from an underlying assumption 
that if certain conditions judged to be important for an effective program are 
present, the output from the program will be satisfactory. The presence of 
qualified teachers, up-to-date equipment, a good curriculum, et cetera would, 
it has been assumed, produce graduates who have marketable ertry level skills 
and who will make desirable employees, while schools lacking in these attri- 
butes will produce less competent and less desirable employees. Actually, 
without extensive follow-up of students there is no way of knowing whether 
there is any important relationship between the process variables and the 
real life criterion of or-the-job effectiveness. Therefore, it is important 
to know what happens to students (preferably all of them) after they leave 
school as dropouts or with diplomas. Follow-up should provide information 
for at least three and preferably five years from course completion. 

This report and the full report that preceded it have been prepared 
with the intention of avoiding excessively negative criticism based upon the- 
oretical considerations for which there are now no expedient means of imple- 
mentation. For example, proven coppetency-based assessment devices for state- 
wide evaluation of student performance do not now exist. Nor has any state 



*This document summarizes findings and recommendations from a much * 
larger report prepared by Educational Testing Service titled, A Study of 
the Efficiency and Efficacy of Evaluation Practices of the Illinois Divi- 
sion of Vocational and Technical Education - Final Report. The names of 
persons who worked on the study and recognition for the commenuable co- 
operation granted by staff members of the DVTE are noted in the full report. 



or local education agency (LEA) successfully developed and installed a fully 
comprehensive system.' Therefore, while it is necessary to keep firmly in 
mind that the Three Phase System is critically deficient when compared 
against ideal standards and to avoid encouraging its users or those who learn 
about it to conclude it can be retained or adopted without need for constant 
revision and adjustment, the syste.1 must not be excessively criticized and 
condemned. 

ThU investigation revealed strong support for continued use of the 
Three , Mse Syste.. Both educators and „o„-ed„cators Judged th. Sy.t^ to 
. valuable and „o„h the e£fo„ that It require.. An important aspect of 
the Syste. Is that it Is designed to produce and deliver large quantities 
of infor^tlon that are useful and necessary for its own .edification; it can 
£ r this reason be rated highly self-reactant and capable of n,odif ication. 

has produced changes In local plans fro- one year to the next, and pos- 
sesses a functional feedback loop that involves the Progra. Approval and 

Evaluation Unit, other units of the T)VTF =,n^ i i ^ 

"-^ °^ the DVTE. and local education agencies. 

Therefore, a major strength of the System is that information is periodi- 
cally obtained that can help evolution toward a more effective form of 
evaluation. 

But the System produces discrepant informtion. In the composite 
ratings that are prepared fro„ site visitations, agency personnel receive 
consistently high ratings at the same tlae that Resources Utilised, „hlch 
.ust he linked to personnel performance. Is negatively related to the finan- 
cial needs of agencies; this suggests that personnel deficiencies may go un- 
recognised by Visitation teams. A consistent negative relationship heLn 
financial needs and ratings sueeests th^t 4<: 

, suggests that, if composite ratings reflect 

local potential achievement... proportionately higher support should perhaps 
given to poorer agencies than is now being allocated, of course, since 
no information exists to establish that local potential Is realistically con- 
s ere when ratings are made, this conclusion cannot be strongly made. In- 
e ecause of lack of information about how these adjustments are made by 
Visitation teams, composite ratings have little established validity. 

nannir^ activities of agencies have n^erous deficiencies. Some defi- 
ciencies ^y arise In part because documents prepared by the Division to guide 
Planners lack the kinds of specificity and elaboration that would make planning 



easier. However, other types of planning assistance are also needed. Half 
or more of the respondents to a questionnaire distributed in this study agreed 
or strongly agreed that their planning activities would have been more success- 
ful had they been given additional external help to: 

1. Conduct student follow-up studies. 

2. Conduct needs surveys in their communities. 

3. Conduct employer need surveys. 

4. Prepare behavioral objectives. 

5. Conduct effective evaluations. 

6. Identify job entry skills of students. 

7. Set priorities for program development. 

The Division currently provides some assistance in each of these areas 
and cannot be faulted for not being fully helpful in all, but it is obvious 
that a very great amount of assistance is needed by local agencies if they 
are to become more effective in their own evaluation efforts. In particular, 
they will need assistance in performance assessment if comprehensive state- 
wide evaluation is to be highly effective. Local agencies may need much more 
assistance in these areas than has commonly been thought. 

The Three Phase System^ as developed and implemented in Illinois^ does 
not emphasize assessment of the output of programs. Instead, unde'r the tacit 
assumption that if conditions are present outcomes will result, the System 
concerns itself with the contents of plans submitted by agencies and what can 
be learned from site visitations keyed to traditional process concerns. These 
types of information cannot, however popular the System may be at this time, 
frlrmly establish the value of programs to their graduates. 

Development of a fully comprehensive evaluation system has not been 
achieved in Illinois. Cooperation between the different agencies is still 
needed and coordination is required. This coordination should, it must be 
noted, extend beyond the boundaries of individual states and incorporate con- 
cern for what states can do together. Therefore, just as the Illinois System 
has already benefited from information developed elsewhere and is currently 
inviting future multi-state efforts, the existing major need for comprehen- 
sive system development can best be met by formally cooperative development 
with other groups. 



INFORMATION NEEDS OF THE DIVISION OF 
VOCATIONAL AND TECHNICAL EDUCATION 



ERIC 



The DVTE is organized under the Board of Vocational Education and 
Rehabilitation through which it is related to the Governor's Advisory 
Council. It has a director and two assistant directors having responsi- 
bility for the performance of eight operational units. An Administr.^cive 
Planning Council made up of the eight Unit Coordinators relates to the 
Office of the Director. The Superintendent of Public Instruction is des- 
ignated by statute to be the Executive Officer for work of the Division. 

To understand the relationship between Division needs for informa- 
tion and how the Three Phase System works, it is necessary to consider 
the goals which were set for the Evaluation System: 

1. To promote and assist with the development of luality 
occupational education programs at .the local level. 

2. To provide the Division of Vocational and Technical 
Education with data upon which better statewide plan- 
ning of occupational education programs can be based. 

3. To help assure accountability of Federal and State funds 
allocated to local occupational education programs. 

These goals suggest how evaluation activities within the Division and in 
the field are intended to fit together to produce a coordinated planning 
and evaluation sequence. The functional parts of the Three Phase System 
are described by the Division as follows: 



Phase I; 



Utilization of self-evaluation data in program 
planning by the local district culminating i^T 
the development of a One and Five-year Plan. 

Program approval by staff of the Illinois Divi- 
sion of Vocational and Technical Education. 



Phase III! 



External e valuation by a visiting team to de- 
velop a profile of the local urogram with con- 
clusions and recommendations to the local district. 



During Phase I. people in local educational agencies receive de- 
scriptions of the System and guidelines for the submission of planning 
documents. During Phase II. LEA plans are reviewed for approval. The 
local plan, after approval by the district and its local governing board. 



is transmitted through the regional Superintendent of Educational Service 
or the Illinois Junior College Board to the Program Approval and Evaluation 
Unit of the Division. This Unit, with assistance from other persons, eval- 
uates the Local Plan and recommends that it be given one of three approval 
levels: Fully Approved (A), Conditionally Approved (CA), and Not Approved 
(NA). 

Site visitation is the critical core of the Three Phase System. Aft^r 
the selection criteria that are used at a given time have been applied, an 
agency to be visited is notified. This notification sees in motion a series 
of activities that end in the mailing of an evaluation report to the agency 
that was visited. 

Phases I and II are annual processes while Phase III occurs every 
two to five years or as warranted. The outcome for Phase I is a document 
which serves as a contractual agreement between the Division and the Local 
Education Agency. In Phase II, the outcome is approval status given to 
each occupational course as well as overall approval of the local plan by 
the state director. The outcome for Phase III is an evaluation report with 
a projected profile of the local district's total occupational education 
program. Additional operational details for implementing the Three Phase 
System consist of pre-visitation preparation including composition of the 
visitation team, preparation of the evaluation report, ::;nd format for a 
local plan. 

The Illinois Division of Vocational and Technical Education is, by 
its control and function, an agency that most often responds to goals and 
objectives set elsewhere rather than establishing its own goals. The Divi- 
sion must, if it is to receive Federal funds for disbursement, respond to 
Federal laws with its specified plans and review the plans submitted by 
local educational agencies. The Division must also respond to State of 
Illinois requirements for the funds its legislature provides, to the advice 
of formally organized advisory councils, and to innumerable other concerns 
for guidelines from educational agencies, groups, and individuals. 

The Division also responds to a large number of internally initiated 
objectives appearing in various documents including: A Systems Approach for 
Management of Vocational Education^ and the Fiscal Year State Plans for the 

^Revised September 1, 1971 (Working Draft for discussion purposes only) 



Administration of Voaatiomil and Technical Education in Illinois.^ These 
objectives, when taken in combination with those that arise outside the 
Division, produce a pool of many hundreds of objectives which can be con- 
sidered as one tries to learn what the Division is attempting ,:o accomplish. 

The Division produces many thousands of copies of other forms and re- 
ports among which are requests for proposals to do funded work; procedural 
guidelines for the submission of requests for funds and for reimbursement 
of programs; and program and project descriptions. Some reports are of 
general information value; other, are for use by very limited audiences. 

For the sake of clarity, a way of combining objectives was adopted by 
Educational Testing Service staff members that called for summary information 
needs statements to be prepared from statements of goals, objectives, and so 
forth, present in the very numerous documents. Furthenr.ore, information needs 
statements were sought that would describe vocational education outcomes ex- 
ternal to the Division (that is, within and about local education agencies) 
and those which might be needed to describe internal Division functioning. 

Educational Testing Service staff members, after reviewing the mate- 
rials available to them, prepared a summary set of 13 objectively stated in- 
formation needs requirements against which the performance of the Three Phase 
System could be compared. (See Column 1 in Table 1 which follows.) Table 1 
briefly summarizes relationships" that exist between Three Phase System oper- 
ations and the external and internal information requirements of the DVTE. 
In the two right hand columns of Table 1, brief statements about the extent 
to which answers are provided can be read and, for each, a statement of new 
procedural requirements that may be needed to improve the System have been 



made. 



Study of the contents of Table 1 will reveal that the Three Phase 
.ystem is designed to meet almost all information needs of the DVTE Also 
the System provides a large amount of information for the Division. Recom- 
mendations on the following pages suggest how the System, which is a model 
of its type, might be further improved. 



2p annually by the State of Illinois, Board of Vocational FH,. 

cation and Rehabilitation, Vocational and Technical Education SvisJon. 
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LONG RANGE RECCMMENDATIONS 



The Three Phase System has as Its major deficiency lack of Infor- 
matlon about job entry performances of students vho complete occupa- 
tional programs. This weakness exists at the center of a cluster of 
closely related information deficiencies all of which urgently require 
attention if statewide ;5valuation is to be made comprehensive. What 
jobs are available? What placement successes have students had? What 
skills are needed to perform on jobs? How satisfied ar>. students 
and employers with the preparation provided in occupational programs? 
All of these questions have concern for the outputs of programs. Lack 
of success at assessing outcomes is the primary deficiency of the Three 
Phase System. Ultimately, evidence of program effectiveness must be 
sought and reasons for program planning and modification must be 
assembled. This will require long range development in the three areas 
that follow; 

1. A fouT'-yeax* plan for performance assesment should be initiated. 

The primary deficiency of the Three Phase System is that it has 
lacked really productive concern for job entry skill assessment and for 
questions which relate to It. A very good reason for not having an 
operational performance assessment program is that its development costs 
would have far exceeded those for the Three Pha^e System. However, at 
the same time that some task analysis projects have been initiated in 
Illinois»many other performance assessment projects have been started 
about the country. Project Career in Massachusetts is illustrative. 
Illinois should actively involve itself in these types of development. 

The selection of a four year cycle for performance development 
assessment by the DVTE is arbitrary but meant to make kn^*m the minimum 
amount of time that ma^* be required to develop such an approach. 

The evolutionary development of a new output-*focusedL evaluation 
system for the Division might » it must be noted » follow along lines 
where site visitations become more and more keyed to output assessment. 
For example » two years from now visitation teams might become involved 
in the field trial of performance assessment studies. This type of 
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development would require collaboration with external agencies. It is 
for this reason that establishment of a Technical Advisory Committee is 
recommended as part of the short range actions in the next section. No 
single state can afford the expense of developing this type of system. 
Indeed, even a consortium of states and agencies will have difficulty 
doing so. Therefore, it is critically important that the Division seek 
to avail itself of all information and opportunities open to it. Col- 
laborative development with e^hcr ape.'.rie> ■.--■.(.rra tn he. -u^ i 

' ^T'^'- lY^i^" "^s'T.s to be the only approach 

that can keep costs within the limits'that exist in Illinois. 

2. A oontinuous effort should be made to improve supply-and- 
demand sti'dies. 

The Three Pha^e System has been designed with the assumption that 
local education agencies should undertake much of the work necessary to 
determine what opportunities will exist for its students and to achieve 
a fit between the opportunities thnt exist and the educational piograms 
designed for them. It is true that local agencies are expected to use 
information given them by the employment services and other groups. 
But the information those groups have provided to date is not good 
enough to permit the local agency to closely tie its curriculum to 
opportunities. 

This problem is also one that is too great for local agencies and, 
perhaps, even for states to meet. It can easily be seen that patterns 
of industrial plant development, changes in consumer demand, and na- 
tional and international economic development forces are far beyond the 
capabilities of restricted areas to understand or sense. These influ- 
ences produce demands for regional, state, and national study. They 
also require high levels of cooperation among agencies and competent 
long range planning. Clearly, the potential effectiveness of the Three 
Phase System has been severely restricted by lack of this type of infor- 



mation. 



3. Formal, contiming efforts should he made to improve oomprehen- 
s%ve evaluatzon and development. ^ 

Comprehensive planning for statewide evaluation of programs may be 
initiat«^d through the use of a half-dozen guidelines. The guidelines 
will seen: IHealized and impossible of attainment in the short range. 



ERIC 



10 



Indeed, they may be so general as to discourage some groups from attempt- 
ing to approach them. However, their intended purpose is not to set 
standards for immediate attainment but rather to help planners to estab- 
lish long range goals. Six guidelines follow: 

Guideline I : Responsibility for establishing statewide goals is shared 
by many groups within the st; Any differences in either philosophi- 
cal or programmatic goals should be resolved as part of initiation of a 
system of statewide evaluation. 

Guideline II ; If evaluation is to be a continuous process, it must be 
built into the fabric of the state administrative structure. This means 
that an appropriate staff organization with funds necessary to function 
must be provided. 

Guideline III : Since evaluation should be a continuing process, not a 
one-shot endeavor, budget provisions should be made on a continuing basis. 

Guideline IV : A comprehens:-'*^ plan for statewide evaluation should show 
(a) major and minor objectives; (b) concern for needs of all the institu- 
tions, agencies and offices involved; (c) proposed evaluation processes 
and procedures, complete with a detailed accounting of the purposes for 
evaluation components; (d) the way each evaluation component will work; 
and (e) how evaluation data will be used. 

Guideline V : If we accept the idea that good evaluation is a dynamic 
continuing process, it is imperative that feedback mechanisms for Improv- 
ing both the evaluation system and the statewide vocational technical 
education system it purports to analyze are built into the system. 

Guideline VI : The worst thing that can happen to a carefully developed 
evaluation report is to have its recommendations ignored or lost in 
bureaucratic processes* Whenever an evaluation system is designed, it is 
very important that the ? esults of the evaluation will be transformed in- 
to specific plans of action by specific offices of the State and by Local 
Education Agencies. Therefore, the Three Phase System must have input in- 
to the total planning system for Vocational-Technical Education in 
Illinois. 
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SHORT RANGE RECOMMEND AT lONi*. 



Both long and short range reconnnendaticT.3 are made to be imple- 
mented as promptly as possible. The difference is that short range 
recommendations may be met within a year while long range recommenda- 
tions cannot possibly be attained in that time. Seven short range 
recommendations f ■^llow. 

1. Establish a teohnioal advisory panel to advise on the evolu- 
tion of a more output-a entered system. 

The committee might have a half-dozen members with most from out 
of state and involved in state level evaluaTion. The charge of the 
committee would be to recommend changes in the current system through 
research and development. 

2. Fiscal audits should be rapidly pursued so that agencies will 
no longer experience as many disarepanoies as have appeared 
^n the past. 

Fiscal audits have not caught up with current operations. Also, 
the type of audit that is undertaken is not sufficient to establish a 
sound financial basis for the pooling of resources and allocation of 
money toward needy areas. It has served to make agencies aware of 
their responsibilities for financial accountability and has' revealed 
that procedures in use before adoption of the Three Phase System were 
inadequate. 

3. Profile analyses should be conducted for each new round of 
evaluation reports. 

Composite ratings for personnel lack variability although no 
evidence arose from analyses of profile data to suggest systematic 
biases were induced by team leaders. Strengths and weaknesses of the 
composite - a ting system are partially assessable by the types of anal- 
yses that were undertaken for this study. These analyses should be 
repeated and improved over time; the Division has shown it is capable 
of doing this. 

4. rje follow-up of students who have been enrolled in programs 
should receive greatly expanded emphasis. 

Follow-up instrumentation_should ask students to evaluate their 
educational experience. Tl{^y should be asked to identify skills they 
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found most valuable to them and to tell which they did and did not 
get from their schooling. Their supervisors should be asked for evalu- 
ations of the performance of former students, especially regarding 
areas In which their training was Inadequate* Employers should also be 
asked to describe the work habits and attitudes of the former student 
as they would compare with those desired In responsible workers. Col- 
lection of this Information should. If at all possible, be made an In- 
tegral, mandatory feature of evaluation with summary Information pro- 
vided to the Advisory Council, the state agencies, and Individual 
schools. 

5. Improvements in the sofpjxxve of the system should he sought. 
In particular, the instructional sections of the Criteria 
and Guidelines documents should be considered for simplified^ 
revision. 

The versions of these documents that were In use during Fiscal 
Year 1972 were brief and could not be easily understood or used. De- 
scriptions of the allocation of funds are, for example, hard to under- 
stand. Improvement of these documents will probably reqalre close 
working relationships with local agencies. 

6. A strenuous effort should be made to overcome the undupli- 
cated-duplioated count problem of student accounting. 

It should be possible to adopt, modify or develop a procedure 
whereby educational agencies would be able to complete but one set of 
records, as for example course registration cards from which both 
state and federal reporting requirements could be met. If necessary, 
alterations should be sought in state and federal reporting require- 
ments • 

7. More formal procedures should be sought for deoision-making 
within the DVTE so that evaluative information can be better 
used. 

Management procedures of the Division have not formally provided 
for inclusion of Program Approval and Evaluation Unit Information in 
the decision-making activities of the Division. For an evaluation 
system to be comprehensive and effective, it is necessary that it be 
formally designed to respond to the information needs of the Division. 
A fully comprehensive system cannot be developed without this type of 
involvement at the Division level nor can existing strengths of the 
present system be most effectively used without it. 
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SUMMARY 



The Illinois Three Phase System should be temporarily retained 
as an administrative tool to be used in relationships between the 
Division of Vocational and Technical Education and local education 
agencies. But it is clear that such a system, with its scope di- 
rected toward local school districts, with its lack of outcome 
assessment, and with its lack of state priorities should not be judged 
a fully satisfactory vehicle for statewide evaluation. It is not neces- 
sary to do additional research to recognize that the greatest need for 
improvement and modification of the System is for improved, fully com- 
prehensive follow-up procedures. Similarly, there is a great need for 
progress in the area of performance assessment for students who com- 
plete programs. 

Because the Three Phase System as it has been developed and 
Implemented is in many respects a model of how to get the greatest 
good from visitation procedures, and because the individuals respon- 
sible for its use have actively sought and built into its design 
procedures by which it can be changed, one cannot recommend that cur- 
rent uses of the System be suddenly and totally changed. A supple- 
mental development program is needed. The development program should 
lead to progressive modification of the present System with the goal 
of ultimately replacing it (perhaps within four years) with a more 
comprehensive output-focused system. Improved performance assessment 
and follow-up should be the cornerstones of a modified system. 
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